Repository navigation
bench: rank file sizes against Parquet in both size groups - #227
Conversation
The Compression Size and SQL File Size groups showed the same metric, file size per format, from two suites. One File Sizes group now charts the compress-bench datasets and three SQL scopes: ClickBench, TPC-H at SF10, and TPC-DS at SF1. Old Compression Size links redirect to it, and the homepage card reads the group by its new or old name. The summary compares each format with zstd Parquet on the compress-bench datasets and on TPC-DS. Each format uses its newest commit that has zstd Parquet. The wide-table and sparse-wide families count as one dataset each. A format measured on only some datasets says how many, has no total, and ranks after the complete formats. Parquet codec variants, such as parquet-snappy, draw in Parquet's color with a dashed line. Signed-off-by: Will Manning <will@willmanning.io>
File Sizes moves to the Compression Size group key, so old Compression Size links open it. SQL File Sizes comes back on its own key: it charts every SQL benchmark and scale factor and ranks formats against Vortex, as before. Each group's info tooltip says how the two differ. Signed-off-by: Will Manning <will@willmanning.io>
Signed-off-by: Will Manning <will@willmanning.io>
A series with no time for a query now ranks below every series that ran all of them. The missing-series penalty alone scores a skipped query at 2x the best time, so a series could rank first by skipping the slowest query. Lance skips TPC-DS q72, which it cannot run. Snappy Parquet joins TPC-DS's queries, so a codec variant is now dotted, and dash-dotted on DuckDB, to keep it apart from DuckDB's dashed Parquet. Signed-off-by: Will Manning <will@willmanning.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🔦 Lighthouse audit
Run |
Vercel adds its Toolbar to preview deployments, and the proxy's x-vercel-skip-toolbar header does not stop it. The Toolbar loads from vercel.live and Pusher, and its third-party cookies and issues cut Best Practices to 0.78 on every audited page. The site loads neither host, so Lighthouse now blocks both. Signed-off-by: Will Manning <will@willmanning.io>
Parquet is zstd level 3, the size and throughput baseline. Parquet (Snappy) is Parquet as DuckDB writes it by default, which TPC-DS SF1 queries and the compression benchmark sizes. ClickBench and FineWeb query their published files as Parquet, and ClickBench's are Snappy. The page also says that a query summary ranks a series that missed a query below the complete ones, as Lance on TPC-DS q72. The contract now names file sizes by format directory, not by codec. Signed-off-by: Will Manning <will@willmanning.io>
vortex-data/vortex#11720 now installs the DuckDB 2.0.0 CLI, the version the queries run on. Signed-off-by: Will Manning <will@willmanning.io>
A new section gives each Parquet writer setting and the test behind it: zstd level, dictionary size, page size and page index, Bloom filters, and row groups. Parquet (Snappy) is now the same Parquet with Snappy in place of zstd, rather than Parquet as DuckDB writes it, and TPC-DS at scale factor 1 no longer queries it. Signed-off-by: Will Manning <will@willmanning.io>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…ize-metrics-0b3056 Signed-off-by: Will Manning <will@willmanning.io> # Conflicts: # lighthouserc.cjs # src/bench/lib/methodology.ts # src/bench/lib/queries.ts
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c0cccc5a6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…cache A format with one wide table or sparse-wide dataset counted the whole family as measured, so it could rank in the complete tier against formats measured on every member. A family now counts as measured only when the format has every dataset of it that any format has. The File Sizes group reuses the Compression Size slug with another name, other charts, and another summary, so the group payload cache moves to v16. Signed-off-by: Will Manning <will@willmanning.io>
…B's file out of its ranking TPC-DS records DuckDB's database file, which is a query engine's storage rather than a file format, so the summary no longer ranks it. The group still charts it. The contract and methodology also say that TPC-DS SF1 records Snappy Parquet's size. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f5eb6dbbaf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The new name keeps more continuity with the Compression Size group it replaces. Signed-off-by: Will Manning <will@willmanning.io>
The group charts Snappy Parquet beside zstd Parquet, so its legends, tooltips, toolbar chips, and summary card name the zstd files parquet-zstd, and the card's baseline header reads "vs parquet-zstd". ClickBench's chart keeps parquet, because it charts the published Snappy files. The series keys stay parquet, so filters, colors, and URLs are unchanged. A compression chart's permalink page uses the same label; a file-size permalink page does not, because the SQL File Sizes group shares those charts. The group payload cache moves to v17. Signed-off-by: Will Manning <will@willmanning.io>
…atio The Compression & File Size card ordered formats by their ratio versus Arrow once every format had one. TPC-DS has no Arrow size, so that order left out the one SQL scope the summary ranks. A size summary now orders by the Arrow ratio only when it covers every dataset each format is ranked on, and by the Parquet ratio otherwise. The String Encoding Size card, whose Arrow ratios cover every dataset, keeps its order. The methodology now says what the vs Arrow column measures, that it covers the compression datasets alone, and how the card is ordered. It also says that the compression benchmark writes 1,048,576-row row groups. The group payload cache moves to v18. Signed-off-by: Will Manning <will@willmanning.io>
On the landing page, the line with the methodology link and the build commit was at the bottom, after every group. It now opens the page's main content, left-aligned with the group cards and small, and the methodology link is underlined so it reads as a link. The chart and methodology pages keep the line at the bottom. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f7d36b5b26
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
In the Compression & File Size group, the toolbar's parquet-zstd chip also toggles ClickBench's parquet line, which is the published Snappy files. A renamed chip's title now says so: "parquet-zstd, and parquet on clickbench". Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a4c669cecf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Rename the SQL File Sizes group to SQL Benchmark File Sizes and move it after the query groups, since it holds the files those queries read. Its summary now compares each format with parquet, bench's standard Parquet, at each format's newest commit that also has parquet. It ranks only JOB, Polar Signals, statpopgen, TPC-DS, TPC-H, and nested TPC-H, whose Parquet bench writes with its own settings; ClickBench, FineWeb, Appian, sorted ClickBench, and the Vortex queries keep their charts. Each benchmark counts once, as the geometric mean of its scale factors, so TPC-H's three scale factors no longer outweigh the rest. DuckDB's database file is left out, as in the Compression & File Size summary. The card drops the Total and vs Arrow columns, which this summary never filled, and says "of N benchmarks" in its coverage note. With no baseline, the landing page's headline returns for this group. Signed-off-by: Will Manning <will@willmanning.io>
The homepage's file size panel dropped bench's coverage for that one panel, so a format without TPC-DS sizes, such as Lance today, showed a geomean over fewer datasets beside formats measured on all of them. Lance is about 2x Parquet without TPC-DS and 4.3x on it, so the panel flattered it. Pass the coverage through, as the other panels do; results.js shows such a row as "Not measured". Signed-off-by: Will Manning <will@willmanning.io>
…factors The SQL Benchmark File Sizes card counted a benchmark only when a format had every scale factor of it, so Lance, with TPC-DS at SF1 alone, showed a 2.12x score beside "measured on 0 of 6 benchmarks". It now counts one scale factor as enough, so Lance shows 1 of 6 and still ranks after the formats measured on every benchmark. The Compression & File Size summary keeps the stricter rule for its families. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2e0d8959f9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
DuckDB's database file is the file DuckDB's queries read, so the SQL Benchmark File Sizes summary keeps it when it has a size on every benchmark the summary covers. With fewer, it stays out, as in the Compression & File Size summary. Signed-off-by: Will Manning <will@willmanning.io>
The collapsed group's size headline read the Vortex ratio alone, so a mean over fewer datasets than the summary covers showed as a plain "smaller than Parquet" claim while the card marked it partial. The headline now adds "(partial)" when Vortex's measured count is below the total. Signed-off-by: Will Manning <will@willmanning.io>
This changes how the benchmarks write Parquet, and which Parquet sizes they record. ClickBench and FineWeb still query their published files as `parquet`. Before, each suite set its own Parquet writer settings. statpopgen and SpatialBench's zone table wrote uncompressed Parquet, and the `vortex` suite wrote Snappy. Now every generated Parquet file uses one config, in `vortex-bench/src/utils/parquet.rs`. It sets zstd level 3, dictionaries of at most 64 KiB, a page index, and pages of at most 20,000 rows and 1 MiB. Each file records a layout version, and the generators write a cached file in an older layout again. Before, TPC-H and nested TPC-H wrote row groups of 1,048,576 rows on all storage, and TPC-DS wrote 122,880. Now every query suite writes 122,880 rows (DuckDB's default) for local NVMe and 1,048,576 rows (parquet-rs's default) for S3. In a TPC-H test at scale factor 10, the small row groups were 28% to 38% faster on NVMe. On S3, the large row groups were 14% to 21% faster on DataFusion, and 3% to 8% slower on DuckDB. Each other setting was tested on its own, on TPC-H at scale factor 10 and on the random access datasets. 1 MiB dictionaries made files 22% larger, DataFusion 8% slower, and lookups up to 2.8 times slower. Bloom filters made files 14% larger and sped up no query. The 20,000-row page limit made lookups up to 4 times faster at the same size. compress-bench and TPC-DS at scale factor 1 now also record the size of `parquet-snappy`: the same Parquet as `parquet`, with Snappy in place of zstd. No query reads it. TPC-DS at scale factor 1 also records the size of Arrow IPC and queries Lance on DataFusion. Lance skips q72, which overflows its 32-bit string offsets. Parquet sizes and times change for every suite that generates Parquet. The bench side is vortex-data/vortex.dev#227. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Will Manning <will@willmanning.io>
The Compression & File Size group labeled its two Parquet series parquet-zstd and parquet-snappy. They now read parquet (zstd) and parquet (snappy), in the legends, tooltips, chips, and the summary's "vs" header. ClickBench's chart still shows plain parquet for its published Snappy files. A renamed chip's title now lists, for each renamed format, the charts that keep that format's real name, so the parquet (snappy) chip does not claim ClickBench charts parquet-snappy. Signed-off-by: Will Manning <will@willmanning.io>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9fceee30e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…y benchmark In the Compression & File Size group, the collapsed headline and llms.txt said "Parquet" although the summary divides by zstd Parquet and the group also charts Snappy Parquet. They now say "Parquet (zstd)". llms.txt described the SQL Benchmark File Sizes score as a geometric mean over datasets, and its range as across datasets. The score now says each benchmark counts once, as the geometric mean of its scale factors, and the range is across benchmarks and scale factors. Signed-off-by: Will Manning <will@willmanning.io>
What changes
This changes bench's size groups, its landing page, and the file size panel on the vortex.dev homepage. Old Compression Size and SQL File Sizes links still work.
Before, Compression Size ranked formats on the compress-bench datasets alone. Now Compression & File Size ranks every format against zstd Parquet on those datasets and on TPC-DS at scale factor 1 (SF1). It uses the newest result of each format, and leaves out DuckDB's database file. In this group only, the two Parquet series show as
parquet (zstd)andparquet (snappy).Before, SQL File Sizes ranked formats against Vortex, counted TPC-H four times, and showed an Arrow column with no values. Now SQL Benchmark File Sizes comes after the query groups and ranks against
parquet. It ranks only the benchmarks that write Parquet with bench's settings, and counts each benchmark once.Before, the homepage panel showed a size score for a format with no TPC-DS sizes. Now it shows "Not measured", as the other panels do, until develop records Lance and Arrow IPC on TPC-DS.
Query summaries rank a series that misses a query below every complete series, so Lance cannot gain by skipping TPC-DS q72. The charts draw Snappy Parquet with dotted lines. The methodology link and build commit move to the top of the landing page, and the methodology page explains the settings of bench's Parquet writer.
The new sizes and the writer settings come from vortex-data/vortex#11720.
How it was checked
bun run check:ci, tsc, the site tests, the bench tests with their Postgres integration suite, the build, and knip pass. No screenshots yet.git commit -s), for the DCO check.bun run check:cipasses.REUSE.toml.🤖 Generated with Claude Code